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Abstract — Inter-symbol interference (ISI) channels with data 
dependent Gauss Markov noise have been used to model read 
channels in magnetic recording and other data storage systems. 
The Viterbi algorithm can be adapted for performing maximum 
likelihood sequence detection in such channels. However, the 
problem of finding an analytical upper bound on the bit error rate 
of the Viterbi detector in this case has not been fully investigated. 
Current techniques rely on an exhaustive enumeration of short 
error events and determine the BER using a union bound. 

In this work, we consider a subset of the class of ISI channels 
with data dependent Gauss-Markov noise. We derive an upper 
bound on the pairwise error probability (PEP) between the 
transmitted bit sequence and the decoded bit sequence that can 
be expressed as a product of functions depending on current 
and previous states in the (incorrect) decoded sequence and the 
(correct) transmitted sequence. In general, the PEP is asymmet- 
ric. The average BER over all possible bit sequences is then 
determined using a pairwise state diagram. Simulations results 
which corroborate the analysis of upper bound, demonstrate that 
analytic bound on BER is tight in high SNR regime. In the high 
SNR regime, our proposed upper bound obviates the need for 
computationally expensive simulation. 

I. Introduction 

Maximum likelihood sequence detection (MLSD) in chan- 
nels with inter-symbol-interference and data dependent time- 
correlated noise is an important problem in many domains. For 
example, in magnetic recording, the statistics of percolation 
and nonlinear effects between transitions ifTTl . ||6l result in 
noise that exhibits data-dependent time-correlation. Recently, 
similar noise models for nanotechnology based probe storage 
have also been developed and the corresponding detectors 
have been found to have significantly improved performance 
compared to the current state of the art |7!|. It is well- 
recognized that a sequence detector designed for an AWGN 
ISI model can have a significant loss of performance if the 
data dependence and time-correlation of the noise is not taken 
into account. 

In the case of finite ISI channels with memoryless noise, 
Forney lO, l?) presented an MLSD solution based on the 
Viterbi algorithm. Upper bounds on the error probability of 
the detector can be derived based on flowgraph techniques 
12, H, Ho). The work of Kavcic & Moura fH considered 
channels with finite ISI and noise modeled by a finite memory 
Gauss-Markov process. The work of [Q, also presents certain 



approaches (see section |V] in HI) for computing an upper 
bound on the performance of the detector. However, their 
technique is not based on flowgraph techniques, and requires 
an enumeration of all error events of relevant lengths and 
an estimate of the corresponding pairwise error probability 
upper bound. We emphasize that an analytical technique for 
estimating detector performance is of great value since it 
allows us to predict the performance at high SNR's where 
simulation can be time-consuming. 

In an ISI channel with additive white Gaussian noise 
(AWGN), the pairwise error probability (PEP) between two 
state sequences can be easily factorized as a product of 
functions depending on current and previous states in the 
(incorrect) decoded sequence and the (correct) transmitted 
sequence. Let S and S be the transmitted and decoded state 
sequences respectively. Then this means that the probability 
that the detector prefers S to 5^, is denoted by P(S^|5) = 
^k=o^i'^k-iT ^k-i) where /i is a function of current state 
and previous decoded states Sj^_^ = {Sk-i,Sk) and actual 
states S'fc_i = {Sk-i, Sk)- Moreover, the PEP is symmetric 
due to the symmetric nature of white Gaussian noise , i.e., 
P{S\S) = P{S\S). Together, these properties allow the 
application of the error state diagram method for finding an 
upper bound on the BER ifTOll . 

In contrast, for the ISI channel with data-dependent Gauss- 
Markov noise (considered in H)), neither of these properties 
hold. The signal dependent and time-correlated noise makes 
the PEP asymmetric. Further the PEP does not factorize in a 
suitable manner as required for the application of flowgraph 
techniques. This makes the estimation of BER for such chan- 
nels, quite challenging. 

Main Contributions: In this paper, we consider a subset of 
the class of ISI channels with Gauss-Markov noise. For these 
channels, we arrive at an upper bound to the PEP that can 
be expressed as a product of functions depending on current 
and previous states in the (incorrect) decoded sequence and 
the (correct) transmitted sequence. The asymmetric character 
of the PEP, i.e., the fact that P(5|5) 7^ necessitates 
an average over all correct and erroneous state sequences. 
We show that this can be achieved using the concept of the 
"pairwise state diagram" Based on this, we present an 



analytical technique for determining an upper bound on the 
BER. Simulations results show that our proposed bound is 
tight in the high SNR regime. 

The paper is organized as follows. Section HI] introduces 
the channel model and describes the corresponding Viterbi 
decoding algorithm. Section |III] presents an upper bound on 
the detector BER. Section |IV] demonstrates simulation results 
that confirm the analytical bounds. Section IV] summarizes the 
main findings of this paper and outlines future work. 

II. Channel model and Viterbi detector 

We introduce the channel model and the corresponding 
detector in this section. A word about notation. In what 
follows, if Zfe is a discrete-time indexed sequence at k*'' time 
instant, the column vector of sequence samples from time 
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instant ki to k2 is denoted by z^f 
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' Zk2[^ where Pig- Channel model with Gauss-Markov noise. 



ki <k2- We will use the notation /(-j-) to denote a conditional 
pdf. The precise pdf under consideration will be evident from 
the context of the discussion. 

A. Channel Model 

Let flfc denote the fc*'' source bit that is equally likely to be 
or 1. The channel output shown in Figure [T] with intersymbol 
interference (ISI) of length / is given by, 

Zk + nfc, 

where y{a\_j) is the noiseless channel output dependent only 
on the / -I- 1 past transmitted bits. The noise Uk is modeled as 
a signal dependent Gauss-Markov noise process with memory 
length L as explained below. 

where the vector h represents L coefficients of an autoregres- 
sive filter, a{a\_j) is signal dependent parameters and Wfc is a 
zero mean unit variance i.i.d Gaussian random variable. Note 
that in the most general model (considered in [I]), even the 
autoregressive filter b would depend on the data sequence a. 
However, in this work, we only work with models where h is 
fixed. We revisit this point in Section [III] The noise Uk can 
be rewritten as. 
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This implies that 



Zk = y{4.i) + P 
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From above analysis, we can conclude that 

f{zk\zt\a) = f{zk\zlzlatL-i), (D 
where we recall that f{-\-) represents the conditional pdf. 

B. Viterbi Detector 

The maximum likelihood estimate of the bit sequence 
denoted a is given by 

d = arff max f(z\a) 
^ae{o,i}" 

= arg max U^-^^^ f{zk\z^-\d) 

= ai"g ^kJofi^k\z^k-h4_^_j) (Using O) 



= arg max 11 
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We define a state Sk = cik-L-i+i (there will be a total 
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of 2^+^ states). With this definition, /(; 
/(-^fc-il'^fc-i)- Moreover it is Gaussian distributed, 

fi4^L\sLi) - N{ji{st,),cist,)) 

where p,{S^_i) is the mean vector and C{S^_i) is the 
covariance matrix. 

With our state definition, we can reformulate the detection 
problem as the following MLSD problem. 

o riN~lf(4~L\'^k-L-l) 

S = arg max n,._r, ; — — ; 

S f{ztlWLL-l) 



arg max Il^-n ; — i ; 

S f{ztl\St^) 
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5 -0 



+ (zLl - -KSLi) fCiSt^)-\zt^ /i(5ti)) 
- [z^zl - fi'iSLi)fc{Stiy'{ztl M'(^ti)) 

where S is the estimated state sequence, c{Sj^_j) is the upper 
L X L principal minor of C(S'^_j) and p,'{S^_^) collects the 
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first L elements of fl{S^_^). It is assumed that the first state is 
known. With the metric given above, Viterbi decoding can be 
applied to get the ML state sequence |9J and the corresponding 
bit sequence. 

The matrix C{S^_i) is of dimension (L + I) x {L + 1). 
For higher values of L, the complexity of detector increases 
as the decoding metric involves the inversion of the matrix 
C{S^_j). However, the matrix inversion lemma can be used 
here to obtain 
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Using Q, we can simplify the detector as follows. 



S = argmin ^ logcr2(a^_^) 



s 
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It should be noted that the above expression does not involve 
any matrix inversion. This reduces the complexity of the 
detector substantially. Another observation is that the Viterbi 
decoding metric involves passing z^_]^ through a filter [—ir 1] 
which is the inverse of the autoregressive filter of noise process 
rife shown in Figure [T] The metric first uncorrelates the noise 
with an FIR filter and then applies the Euclidean metric to the 
output of the filter 

III. Upper Bound on BER 

As discussed previously, the channel model under consid- 
eration (cf. Section inii, is such that the corresponding PEP is 
asymmetric, and moreover does not factorize as a product of 
appropriate functions as required by flowgraph techniques. We 
now show that we can address this issue by using the Gallager 
upper bounding technique ||5l|, coupled with a suitable change 
of variables. 

Denote an error event of length as ejv = {S,S) such 
that S and S are valid state sequences and Sk ~ Sk, Sk+N = 
Sk+N, Sk+j ^ Sk+j for 1 < j < iV - 1 and Sk+j = Sk+j 
for other values of j where Sk and Sk are the estimated and 
correct state respectively. Using this, an upper bound on the 
BER can be found as follows |2|, 

oo 
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where i'{S, S) is the number of erroneous bits along the 
sequences S and S and En is the set of all error events cat 
of length N. The number of erroneous bits is given by 

where S{ak, flfe) = 1, if afc 7^ cik and Z is a dummy variable. 
Using this the upper bound above can be expressed as 
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where P{S) = P{Sq)P{Si\So) ■ ■ ■ P{Sn\Sn-i 
S is valid state sequence , (M is the number of states). The 
upper bound on the PEP can be using Gallager's technique ID 
as shown below. Let A{S, S) = {z : f{z\S) > fiz\S)}. Note 
that using previous arguments, we also have that A{S, S) 

7v-i/(^*=l-sLi.2.':i) 



P{S\S) - P{So. 



> ly Now, 
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where < pfc < 1 for fc = 0, . . . , iV - 1. 

The above integral can be simphfied as follows. 
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where Uk = [-&'^' 1] • [zk-L ■■■Zk\ 
At(S'^_i),OT(S'^_J) = [-F 1]-^(S'^_J. The Jacobian matrix 
for the change of variables has determinant equal to 1, since 
the corresponding matrix of partial derivatives has ones on 
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the diagonal and is lower triangular. Note that the change of 
variables decouples the original expression, so that it can be 
expressed as the product of N independent integrals. Now we 
can simplify the PEP as follows. 



P(a|a)<minn^-oi / 



exp( 



{i-Pk){uk-mist,)r 



)duk 



exp( 



)duk 
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exp( 



il-p,)a^{a'l_j)+p,a^{4_j) 
(1 - Pk)a^{atj)M'iSti) + PkCTHati)m'iSti] 



2a2(at,)a2(atz) 
((1 - Pk)iJHal_,)m{St^) + Pk<r\al_,)m{Sl^,)Y 



nf=o'^(^ti.^ti) (4) 



where W^C-SfcLi, is a function of (7(a\_j), <7{a\_j), 

9K(S'^'_j) and OT(S'^_j) and the simplification of the integral 
in (|3]l is given in the Appendix. 

It is important to note that the factorization of PEP given by 
dill for our channel model is possible because the autoregres- 
sive filter h is not dependent on the input bit sequence. In 11], 
h is assumed to be data dependent given by h{a\_j). When 
the autoregressive filter h{a\_j) becomes data dependent, it 
is very difficult to write PEP in the form given in (|4]i. In this 
case, the inverse of the autoregressive filter of the noise process 
rife ([— &(a^_j)^ 1]) is state-dependent which means that the 
actual state transition {S^_i) and estimated state transition 
{S^-i) have different filters. In this situation, the specific 
change of variables used above does not seem to work. 

Probability of bit error can now be further simplified as ||2|, 



< A "S^ J_ J_Sr\^T]N-lyS(akAk) 

- dZ ^ 2^2^^^fe=o ^ 
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N=l s I 

For obtaining T{Z), we construct a product trellis. Consider 
a matrix V{Z) of order Af^ x AP, where each row and 
column is indexed by a pair of states corresponding to the 
actual and the errored states. Let represent a state that takes 
one of 2^+^ values. Consider the entry of V{Z) indexed by 
((ft,/3,),(/3^/3^)), 



[y{z)]m,p,)X0[,p]) 

1 0, if either Pi -> P[ or ji^ ^ not allowed 

where W{{l3i, (3j), (/3-, /3^)) can be found for state transitions 
Pi Pi and Pj — 7> P'j using (01 and and aj are latest 
bit in states and /3j respectively. A product state is called 
good state if P'i ~ P'^ and bad otherwise. V{Z) will have a 
structure which has Vgg{Z) (good to good state transition), 
Vgb{Z) (good to bad state transition), Vbg{Z) (bad to good 
state transition) and Vbb{Z) (bad to bad state transition). 



V{Z) 



Vgg{Z) Vgb{Z) 
Vbg{Z) Vbb{Z) 



where the order of Vgg{Z) matrix is M x M and the order 
of Vbb{Z) is {AP - M) X {AP - A/). Now we can write 
T{Z) as, 

T{Z) = a{Z) + b{Z){I - VBB{Z))-h{Z) 

where a{Z) = 1'^Vgg{Z)1, h{Z) = 1'^Vgb{Z) and c{Z) = 
VBG{Z)t. The symbol 1 denotes a vector all of whose entries 
are 1 and / is identity matrix of order (AP — AI) x {A'P — M). 
Using the above result, we can compute Pfc(e) as |2l, 

n(e) < jjla'il) + b'^m - Vbb{1))-'c{1) + b^(l) 
• (/ - VBB{l)r'c'{l) + - VBBil))-' 

■ vum-VBB{i))-h{i)]. 
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dZ 
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^ E j^-^T.T.^t-o'z^'''-'^^pm)\z.. 
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For our model, Vgg{Z) is not a function of Z which means 
that o'(l) = 0. Similarly, c{Z) is also not a function of Z 
which implies c'(l) = and it should also be noted that 
b'-^(l) = The new bound for our channel model is, 

n(e) < ^[b'^m Vbb{1))-'c{1) + b^(l) 

• (/ - VBB{i)r'vum VBB{i))-h(i)]. 
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Fig. 2. BER with different SNR for the channel model with a) 
decoding and b) 16 states in decoding. 



IV. Simulation Results 

In the first set of simulations, we used the following 
parameters: L = 2 with 6 = [.1 .5] and ISI memory / = 1. 
The signal dependent noise variance for 4 states are given 
by cr2(00) = 1, (7^(01) = 2, cr2(10) = 3 and (7^(11) = 4. 
The number of states in decoding is equal to 8 in this case. 
The SNR is defined as signal energy in y{a'^_j) divided by 
total noise variance. We have used a linear signal component 
given as y{a'^_i) = c{2ak + ak-i) where the value of c can 
be varied to change the SNR. In Figure |2l the analytic bound 
follows the simulation BER. At an SNR of 21 dB, the analytic 
bound gives a BER equal to 3 x 10^' whereas simulation 
BER is equal to 2 x 10^'. The analytic bound is quite tight in 
high SNR regime. In another simulation, we used following 
parameters, L ~ 3 coefficients of an autoregressive filter is 
given by 5 = [.1 .3 .5], ISI memory (/) is equal to 1 and signal 
dependent noise variance for 4 states are given by cr^ (00) = 1, 
0-2(01) = 2, cr2(10) = 3 and 0-2(11) = 4. The number of 
states in decoding is equal to 16 in this case. In Figure |2] 
the analytic bound again follows the simulation BER for this 
channel model with modified channel parameters. At an SNR 
of 20 dB, the analytic bound gives a BER equal to 7 x 10"^ 
whereas simulation BER is equal to 4 x 10^^. 

V. Conclusions and future work 

We considered the problem of deriving an analytical upper 
bound for ML sequence detection in ISI channels with signal 
dependent Gauss-Markov noise. In these channels the pairwise 
error probability (PEP) is not symmetric. Moreover, it is hard 
to express the PEP as a product of appropriate terms that 
allow the application of flowgraph techniques. In this work, 
we considered a subset of these channels, and demonstrated an 
appropriate upper bound on the PEP. Using this upper bound 
along with pairwise state diagrams, we arrive at analytical 
BER bounds that are tight in the high SNR regime. These 
bounds have been verified by our simulation results. 



It would be interesting to examine whether our current 
techniques can be extended to address the general channel 
model. Moreover, it may be possible to reduce the complexity 
of evaluating the bound by reducing the size of the product 
trellis by exploiting channel characteristics. We are currently 
investigating these issues. 

Appendix 

The integral in the equation (O can be expressed in the 
following form, 

=^ exp( — {a{x — m)2 + f3(^x — rh)^))dx 
27r7 2 

am^ + /Sm^ (am + 
exp( + -^^T—T^^) 
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(i-Pfc) 
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2{a + p) 
m = M(5ti), 7 



<^^^'"'{al_j)crP>'{al_j) and m = m{S^_^). Using the above 
equality, we can easily simplify the RHS of equation Q as, 
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oi-P'^(at,)'T'"'(at,) ^(i-p,)a^al_,)+pka^al_,) 

((1 - pu)a\di^jmst^) + Pk'rH4-imsLi))' 

2a2(at,)a2(atz)((l - Pk)aHal_,) + Pki^at,)) 
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